Apparatus and method for implementing multiple high speed switching fabrics in an ethernet ring topology

ABSTRACT

A network device for implementing a high speed transmission protocol. The network device includes a plurality of high speed modules which are connected by a plurality of high speed links. Each of the plurality of high speed modules implements the high speed transmission protocol and each of the plurality of high speed modules being implemented in an Ethernet ring topology. A medium access control layer in each of the plurality of high speed modules is modified in a manner that is transparent to a switching entity in each of the plurality of high speed modules. The high speed transmission protocol comprises a plurality of aspects including an in-banding messaging mechanism for efficient and responsive traffic management and network operation.

This application claims priority of U.S. Provisional Patent ApplicationSer. No. 60/762,111, filed on Jan. 26, 2006. The subject matter of theearlier filed application is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a switching protocol in a packetswitching network and more particularly to a system and method ofproviding a high speed protocol for switch devices in a packet switchingnetwork.

2. Description of the Related Art

A packet switching network/fabric may include one or more networkdevices, such as an Ethernet switching chip, each of which includesseveral modules that are used to process information that is transmittedthrough the device. Specifically, each network device includes aningress module, a Memory Management Unit (MMU) and an egress module. Theingress module includes switching functionality for determining to whichdestination port a packet should be directed. The MMU is used forstoring packet information and performing resource checks. The egressmodule is used for performing packet modification and for transmittingthe packet to at least one appropriate destination port. One of theports on the device may be a CPU port that enables the device to sendand receive information to and from external switching/routing controlentities or CPUs.

One or more network devices in a switching fabric may include one ormore internal fabric high speed ports, for example a HiGig™ port, inaddition to one or more external Ethernet ports, and a CPU port. Thehigh speed ports are used to interconnect various network devices in asystem and thus form an internal switching fabric for transportingpackets between external source ports and one or more externaldestination ports. As such, the high speed ports are not externallyvisible outside of a system that includes multiple interconnectednetwork devices. The current high speed transmission protocols for thesehigh speed ports, however, have become an architectural bottle neckbecause they do not scale well with the requirements from higher endsystem designs. For example, the current high speed transmissionprotocols support eight classes which are not enough to differentiatesystem control and network application traffic within the switchingfabric. Current high speed transmission protocols also support up to 128modules which is insufficient for higher end system design andexpansion. In current high speed transmission protocols, the support of4K identifiers in each of the layer 2 multicast and IP multicast spaceis not enough, in some cases, and the hard separation of layer 2multicast, IP multicast and broadcast spaces makes it inflexible tore-allocate limited table resources to meet requirements from differentcustomers' system designs. Furthermore, the design of the headerstructure of the current high speed transmission protocols preventssustainable development. In addition, important information is missing.For example, missing from the current high speed transmission protocolsare load balancing information which enables every port of a switchingfabric to have its own packet parsing logic and a fine granular linklevel flow control mechanism for optimal operation required by higherend fabric designs.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this specification, illustrate embodiments of the invention thattogether with the description serve to explain the principles of theinvention, wherein:

FIG. 1 illustrates a packet switching fabric 100 in which an embodimentthe present invention may be implemented;

FIG. 2 illustrates aspects of the inventive speed transmission protocol;

FIG. 3 illustrates an embodiment of a high speed packet 300 implementingthe inventive high speed transmission protocol;

FIG. 3 a further illustrates an embodiment of a fabric routing controlportion;

FIG. 3 b illustrates one embodiment of a packet processing descriptor308;

FIG. 3 c illustrates another embodiment of a packet processingdescriptor 308;

FIG. 4 illustrates an embodiment implementing pre-emptive transmissionin which in-band messages are transmitted with and among multiplepackets over a high speed link;

FIG. 4 a illustrates the general format of each high speed transmissionprotocol message;

FIG. 4 b illustrates multiple devices which initiate/terminate linklevel messages;

FIG. 4 c illustrates an embodiment of a switching network in whichend-to-end messages are transmitted;

FIG. 4 d illustrates an embodiment of a network implementing moduleregister/table access messaging;

FIG. 5 illustrates an embodiment of the invention in which a switchingfabric includes multiple switching fabrics and multiple devices; and

FIG. 6 illustrates an access component of each of the switching modulesin a ring topology.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Reference will now be made to the preferred embodiments of the presentinvention, examples of which are illustrated in the accompanyingdrawings.

FIG. 1 illustrates a packet switching fabric 100 in which an embodimentthe present invention may be implemented. Packet switching fabric 100uses inventive high speed links 101 a-101 x, implementing an inventivehigh speed transmission protocol which is intended to form acommunication and transport backbone among switching components such as,multiple switching elements 102 a-102 d, multiple traffic managers 104a-104 x, multiple packet processors 106 a-106 x and multiple mediaaggregators 108 a-108 x. Each switching element 102 is a switchingdevice/module on which packet switching fabric 100 is constructed. Itshould be noted that a packet switching fabric 100 may include one ormore switching elements 102. Each traffic manager 104 is a functionalblock/module for handing packet buffering, queuing, scheduling,congestion management and flow control, as well as traffic splicing andshaping functions. Each packet processor 106 is a device for handlingpacket parsing, classification, layer 2/layer 3 (L2/L3) switching, aswell as packet modification and replication functions. Each mediaaggregator 108 is a device for handling the packet transmission on thenetwork through one or multiple ports.

In an embodiment of the invention, each of switching elements 102,traffic managers 104, packet processors 106 and media aggregators 108may take different forms of functionality and device level integrationbased on the performance and cost factor(s) associated with switchingfabric 100. For example, there may be a single switching element 102 inswitching fabric 100. In other cases, multiple switching elements 102may be interconnected in the form of rings or other complex multistagenetworks to form switching fabric 100. However, the inventive high speedtransmission protocol retains its core functionality regardless of theswitching elements 102, traffic managers 104, packet processors 106 andmedia aggregators 108 combinations.

FIG. 2 illustrates aspects of the inventive high speed transmissionprotocol. As shown in FIG. 2, the high speed transmission protocolprovides a transmission link aspect 202, a fabric forwarding aspect 204,a packet processing descriptor aspect 206, an in-band messaging aspect208 and an encoding aspect 210. Transmission link aspect 202 providesfor variable-sized packet based transmission with fixed-sized messagingcapability. Transmission link aspect 202 also providesmessage-over-packet pre-emptive transmission capability (discussed indetail below), and error checking capability for both packet and messagetransmissions. An embodiment of fabric forwarding aspect 204 supports upto 16 traffic class differentiations for packet flows across the system,supports up to 256 addressable physical/logical modules; supportsgeneric multicast forwarding across the system with up to 64 K groups atthe module level granularity and expandable at the port level; supportsexplicit port level indication for physical ports, physical trunks andvarious embodiments of virtual ports/links/channels/tunnels; andsupports explicit fabric design specification operation parameters forpacket-content agnostic fabric operation. Packet processing descriptoraspect 206 provides flexibility for various packet-processing descriptoradaptations, including the existing descriptors developed for currenthigh speed protocols, and provides packet processing flow continuityacross packet switching fabric 100 for system design scalability.In-band messaging aspect 208 provides congestion management protocols,system resiliency protocols, database synchronization protocols andcomponent access protocols. Encoder aspect 210 provides structuredheader design for sustainable developments and is scalable with physicalinterface speed upgrades.

In an embodiment of switching fabric 100 implementing the current highspeed transmission protocol, each component 102-108 has a port levelvisibility across the switching fabric. Each multicast packet sent froman ingress module of one of components 102-108 is sent once and isreplicated to the corresponding set of egress modules which replicatesthe packet further to the corresponding set of egress port(s). Switchingfabric 100 provides for two virtual forwarding planes concurrently, onefor packet transport and the other for in-band messaging. Eachforwarding plane guarantees in-order delivery for traffic with the same{source, destination, traffic class} tuple. An ingress switching fabricmodule and an egress switching fabric module forms a pair of packetprocessing protocol peer which uses packet processing descriptor 206 asthe communication mechanism.

FIG. 3 illustrates an embodiment of a high speed packet 300 implementingthe inventive high speed transmission protocol. Each high speed packet300 includes a control start-of-packet character 302, a controlend-of-packet character 314 which is aligned depending on a length of ahigh speed payload 310, and a control idle character 316 which is usedto fill the gap between high speed packets and/or messages. Each highspeed packet 300 also includes a 16 bit header 304 which carriestransmission header information for the high speed payload 310. Theheader 304 includes a fabric routing control portion 306 which is usedby switching fabric 100 for forwarding operations and a packetprocessing descriptor 308 which is used by elements of switching fabric100 for fine grained traffic management and packet processingoperations. In one embodiment, fabric routing control portion 306 is 7bytes and packet processing descriptor 308 is 8 bytes. High speed packet300 also includes a payload portion 310 for carrying frames, forexample, Ethernet frames. High speed packet 300 further includes apacket error protection field 312.

FIG. 3 a further illustrates an embodiment of fabric routing controlportion 306. As shown, fabric routing control portion 306 includes amulticast field 350 for indicating if the packet is to be unicast ormulticast through switching fabric 100, a traffic class field 352 forindicating the distinctive quality of service that switching fabric 100will provide when forwarding the packet, a destination module identifier354, a destination port identifier 356, a source module identifier 358,a source port identifier 360, a load balancing identifier 362 forindicating a packet flow hashing index for statistically evendistribution of packet flow though the multi-path switching fabric 100,a drop precedence field 364 for indicating the traffic rate violationstatus of the packet as measured by the ingress module, a packetprocessing descriptor type 366 for defining packet processing descriptor308, and multiple reserved fields that are placed between other fieldsof fabric routing control portion 306. When multicast field 350indicates that the packet is to be unicast, destination moduleidentifier 354 indicates the destination module to which the packet willbe delivered and when multicast field 350 indicates that the packet isto be multicast, destination module identifier 354 indicates the higherorder bits of the multicast group identifier. When multicast field 350indicates that the packet is to be unicast, destination port identifier356 indicates the physical port associated with the module indicated bydestination module identifier 354 through which the packet will exitsystem 100 and when multicast field 350 indicates that the packet is tobe multicast, destination port identifier 356 indicates the lower orderbits of the multicast group identifier. Source module identifier 358indicates the source module from which the packet originated. Sourceport identifier 360 indicates the physical port associated with themodule indicated by source module identifier 358 through which thepacket entered system 100.

FIG. 3 b illustrates one embodiment of packet processing descriptor 308.The content of packet processing descriptor 308 fields may varydepending on packet processing flow definitions. In an embodiment of theinvention, different packet processing descriptor 308 overlays may beactive simultaneously over a high speed link 101 and differentiated bypacket processing descriptor type 366. As shown, packet processingdescriptor 308 includes an operation code 380 for indicating theoperation type for the next hop module, a source trunk 382 forindicating whether the source port is a member of a trunk group,multiple mirror fields 384 a-384 x, multiple VLAN identifiers 386 a-386b and multiple reserved fields that are placed between other fields ofpacket processing descriptor 308.

FIG. 3 c illustrates another embodiment of packet processing descriptor308. As shown, this embodiment of packet processing descriptor 308includes an operation code 390 for indicating the packet processinginstructions, a learning enable field 392 for indicating whether thepeer module(s) should learn the MAC source address, a virtualdestination port identifier 394 for indicating a destination virtualtunnel through which the packet is delivered to the network, a virtualsource port identifier 396 for indicating a source virtual tunnelthrough which the packet is received from the networks, multiple virtualswitching identifiers 398 for indicating the packet switching domain andflow classification information which is used to guide switchingoperations and multiple reserved fields that are placed between otherfields of packet processing descriptor 308. In this embodiment, aphysical port is used to indicate the physical network media interface,for example, SGM11 or XAU1 interface. A logical port is used to indicatethe logical network media interface, for example, a SONET channel, aWiFi RF channel or a trunk. A virtual tunnel indicates the logicalpeer-to-peer link across a network path and a virtual switching domainindicates a logical switching plane over which the corresponding policybased switching rules could be applied regarding network scope, routeselection, quality of service policy, etc.

The inventive high speed transmission protocol provides an in-bandmessaging mechanism among devices 102-108 for efficient and responsivetraffic management and fabric operation within high quality packetswitching system 100. Therefore, messages implementing the high speedtransmission protocol may be defined for congestion managementprotocols, system resiliency protocols, database synchronizationprotocols and component access protocols. Each high speed messageincludes a control character, fixed-size message content, and an errorcorrection field. A high speed message may be transmitted over highspeed link 101 alone, or it may be inserted in the middle of a highspeed packet transmission. As such, the inventive high speedtransmission protocol enables pre-emptive transmission.

FIG. 4 illustrates an embodiment implementing pre-emptive transmissionin which in-band messages are transmitted with and among multiplepackets over high speed link 101. Messages 402 a and 404 are transmittedwith packet 412, message 406 is transmitted between packet 412 and 414,message 408 is transmitted with packet 414 and message 410 istransmitted with packet 416. For intra-packet message transmission, forexample messages 402 a, 404, 408 and 410, the message insertion pointswithin a packet transmission are implementation dependent. However, inan embodiment, messages 402 a, 404, 408 and 410 are inserted at theboundary of integer number of 16 bytes of the packet transmission,relative to the control start-of-packet character 302 transmission. Inan embodiment, for inter-packet and intra packet message insertion,back-to-back message transmission, with no idle bytes between messages,are allowed. However, the maximum message burst size is systemimplementation dependent. According to the invention, the in-bandmessaging protocols are designed so that the frequency of messagetransmission do not occupy a substantial amount of link bandwidthresources, such that the regular data packet switching throughputperformance is not affected. The maximum number of message insertionsfor intra-packet transmission may also be limited to the physical designspecifications of the switching system.

FIG. 4 a illustrates the general format of each high speed transmissionprotocol message. Each message includes a delimiter control code 420 toindicate the start of a message transmission, a message protocol type422, a message forward type 424, a message destination identifier 426, amessage source identifier 428, multiple protocol dependent parameters430 a-430 x, and an error correction field 432. An embodiment of theinvention includes link level messages, egress-to-egress/end-to-endmessages and module register/table access messages. The link levelmessages may be used for sending management commands. Egress-to-egressmessage are initiated from a high speed component 102-108 and terminatedby the high speed module peer(s) 102-108. Module register/table accessmessages are designed for a CPU entity associated with module 102-108 toaccess the registers and tables in other modules 102-108 acrossswitching fabric 100 through the in-band messaging mechanism.

The link level messages are initiated/terminated by the Medium AccessControl (MAC) of client peers on both sides of a high speed transmissionprotocol physical or logical link, which may span one or more physicallinks. The high speed logical link is a virtual connection betweenmultiple high speed client peers 102-108. The definition and granularityof the logical link may be system design specific, depending onattributes such as, link traffic classes, sources, destinations orvarious combinations thereof. Some system designs may require aconversion between a high speed physical link and a non-high speedphysical link. Depending on the application, the high speed logicallinks may be mapped to the physical channels on a one-to-one ormany-to-one basis and may be terminated at or tunnelled through theconversion devices, which require the physical channel emulation overthe high speed physical link in addition to the logical link behaviour.

FIG. 4 b illustrates multiple device 440 a-440 d (which may include oneor more of devices 102-108) which initiate/terminate link levelmessages. Device 440 a and device 440 b initiate/terminate link levelmessages 442. Device 440 c initiates/terminates link level messages 448to physical link converter 444 which converts the high speed message tomessages 450 for a non high speed MAC on device 440 d andinitiates/terminates messages 450 to the non high speed MAC on device440 d. Each of device 440 a-440 d also transmits link level messages onhigh speed logical links 446 a-446 x.

Egress-to-egress message are initiated from high speed components102-108 and terminated by high speed module peer(s) 102-108 acrossswitching fabric 100. The message peer definition, message distributionpattern and message transmission quality of service may vary dependingon the protocol and system design. FIG. 4 c illustrates an embodiment ofthe switching network in which end-to-end messages are transmitted. FIG.4 c illustrates egress-to-egress messaging designed for traffic manager104 module to module level transmission flow control at variousgranularities. FIG. 4 c illustrates 3 switching modules, each with oneor more traffic managers 104 and other devices. Depending on theegress-to-egress flow control protocols, messages may be distributed ona one-to-one or one to all basis. Line 402 represents a one-to-onelogical tunnel, i.e., from one traffic manager 104 in module 1 toanother traffic manager 104 in module 1, from one traffic manager inmodule 2 to a traffic manager in module 1 and from one traffic managerin module 3 to another traffic manager in module 1. Line 404 representsthe all-to-one logical tunnel, i.e., all modules to module 1. Based onthe congestion status changes on tunnels to module 1, module 1 maydeliver the corresponding flow control message to all modules toregulate the corresponding traffic.

FIG. 4 d illustrates an embodiment of a network implementing moduleregister/table access messaging. As shown in FIG. 4 d, each of modules460 and 462 is associated with a CPU 468 a/b through a regular PCIconnection and packaged in the format of a management card in a chassissystem. Each of modules 464 a-464 d has no associated CPU entities andare packaged in the form of line cards. Each of modules 460-464 has itassociated management agent logic block 466 to execute theregister/table access commands from a CPU entity 468 and responds withthe results back to the corresponding CPU entity 468. In an embodiment,the message delivery is restricted to peer-to-peer (unicast) onlybetween a CPU entity 468 and a management agent 466 within a module. Inan embodiment, the peer-to-peers (multicast) messaging between a CPUentity 468 and the management agent 466 of multiple modules and thepeer-to-peer messaging among multiple CPU entities are defined asseparate protocols.

According to an embodiment of the invention, multiple switching modulesimplementing the inventive high speed transmission protocol, may beimplemented with multiple devices without the inventive high speedtransmission protocol, wherein the functionality of the switchingmodules implementing the high speed transmission protocol is extended tothe devices not implementing the high speed transmission protocol. FIG.5 illustrates an embodiment of the invention in which a system 500includes the inventive switching fabric 100, multiple switching modules502 implementing the inventive high speed transmission protocol andmultiple devices 504 without the inventive high speed transmissionprotocol. Therefore, the switching functions supported by each device504 are a subset of those supported by switching modules 502. However,this embodiment extends the functionalities of switching modules 502 toassociated devices 504 without increasing the overall system cost. Forexample, this embodiment enables the removal of a CPU subsystem on eachdevice 504, thereby decreasing both the cost and complexity of thesystem design. System level switching delays could also be reduced inthis embodiment of the invention.

Each of switching modules 502 serves as a master entity and each ofdevices 504 serves as a slave entity for its associated switching module502. This allows for in-band messaging, register access and interruptmessages. System 500 also supports in-band link level flow controlmessages. Each of devices 504 supports a IGE wire-speed transmissioncapability and switching modules 502 support 64 logical channels (64port slave designs) per 1 GE uplink for both ingress and egressdirections. In this embodiment, switching modules 502 perform allswitching functions including packet forwarding and filtering, packetmodification and replication, switching protocol implementation anddatabase management, switching level MIB collection and congestioncontrol and traffic scheduling/shaping. Devices 504 perform MAC functionand data-path multiplexing/de-multiplexing functions including MACtransmission and flow control operations, MAC/port level MIB collection,bandwidth oversubscription congestion management, and trafficpolicing/metering. In an embodiment of the invention, local switchingcapability is not required of device 504.

When a packet enters the system, ingress device 504 transmits the userport on which the packet is received and the class of service to whichthe packet belongs to an associated switching module 502. When a classof service becomes congested, switching module 502 transmits informationabout the congested class of service to associated device 504. After thepacket is processed, switching module 502 transmits the user port onwhich the packet should be transmitted to egress device 504 and egressdevice 504 transmits information about congested user ports to theassociated switching module 502. To perform management functions,switching modules 502 send requests for information about registers toaccess for read/write operations and devices 504 return associatedregister access responses. Each device 504 also transmits status changeinterrupts to switching modules 502.

Because each device 504 supports only a IGE MAC, the present inventionlimits the number of fields transmitted in each packet/message. As such,in this embodiment, the header of each packet is condensed from 16 bytesto 8 bytes. A header of a packet/message transmitted through system 500includes a start of logical link delimiter field, a type field whichindicates the packet or control message, a destination identifier forindicating the destination virtual port, a source identifier forindicating a source virtual port, drop precedence field for indicatingthe drop precedence marking of the packet on ingress, an error field forindicating whether the packet is received with an error on ingress and atraffic class field for indicating the traffic class to which the packetbelongs. The header also includes an error correction field which coversfrom the start of logical link delimiter field to the source identifier.The packet includes a payload, for example an Ethernet payload, whichcarries the variable sized packet content starting from the MACdestination address through the error correction fields. The payload mayalso be a fixed sized message content which includes error correctionfields.

In another embodiment of the invention, multiple devices 504 arestackable in a closed/opened ring topology to perform as a single unit.This embodiment allows for in-band messaging for flow control across avirtual “full mesh network.” This embodiment also allows for in-bandmessaging, system management and switching database synchronization.Devices 504 may be stacked in a symmetrical network module, wherein eachdevice 504 of a stack is a standalone switch and a stacking port istreated as just one of the network ports. This allows for minimumstacking header. Devices 504 may also be stacked in an asymmetricalfabric module, wherein each device functions as a combination ofingress-packet processor 106 and egress-packet processor 106 and astacking port are treated as a fabric link. However, it should be notedthat a 1GE uplink may not be fast and robust enough to serve as a fabriclink. This embodiment of the invention allows for a stacking header tocarry additional packet processor index information from the ingressdevice to the egress devices. It should be noted that local switchingcapability is independent of the stacking design model.

According to another embodiment of the invention, multiple switchingmodules 102-108 with up to 10GE wire-speed transmission capability areimplemented in an Ethernet ring topology, wherein the MAC layer ismodified in a manner that is transparent to software L2/L3 switchingmodules. FIG. 6 illustrates an access component 600 of each switchingmodule 102-108 implemented in the Ethernet ring topology. As shown, eachswitching module 102-108 includes dual MAC interfaces 602 that areconsidered as a single trunk interface to the network media. Each MACinterface 602 handles encapsulation and error control for packettransmission. Each switching fabric also includes a copying andstripping control component 604, download queues 606, transition queues608, congestion and topology management entity 610, upload queues 612,and a fair access transmission scheduler 614. Copying and strippingcontrol component 604 filters received packets for packet downloadingand transition forwarding. Download queues 606 queue ingress packets tobe processed by a L2/L3 switching entity. Congestion and topologymanagement entity 610 handles protocols on ring congestion and flowcontrol and ring topology configuration and status change notification.Upload queues 612 queue egress packets from the L2/L3 switching entityand fair access transmission scheduler 614 handles arbitration betweenuploading and transitional packets and steer packets between dual MACinterfaces 602. The inventive Ethernet ring topology 600 offersresiliency and fairness with minimal cost increase and modification overa standard Ethernet interface.

Based on packet flow hashing, each switching module 102-108 randomlyselects a direction on one of dual MAC interfaces 602 on which totransmit each packet. Hence, although the L2/L3 switching entity hashespacket flows among the two interfaces 602, it is agnostic to the ringbehaviour of this embodiment. For peer-to-peer (unicast) forwarding,there is a full-duplex logical link between every pair of ring switchingfabric peers, where the customer MAC/VLAN address learning is associatedwith the logical link. For peer-to-peer multicast forwarding, there is amulti-drop logical link from a ring switching module to all of its ringswitching fabric peers, where tree-pruning is performed at the L2/L3switching level.

Specifically, for peer-to-peer (unicast) forwarding, the L2/L3 switchingentity of an originating switching module 600 decides to forward apacket to another switching module 600 on the ring and hashes todetermine the packet direction on one of interfaces 602. The originatingswitching fabric then transmits the packet to the destination switchingmodule through intermediate switching modules. Each of the intermediateswitching modules 600 passes the packet to the next switching module 600in the transmission path without copying or stripping the packet fromthe ring. When the packet reaches its destination, the destinationswitching module 600 strips the packet from the ring and copies thepacket to its L2/L3 switching entity which switching the packet to oneof its destination customer ports and learns the source customerMAC/VLAN address with the originating switching module. If duringtransmission of the packet, one of the intermediate switching modules600 malfunctions, the originating switching fabric re-steers the packetthrough its other MAC interface 602 to the destination switching module600.

For peer-to-peer multicast forwarding, the L2/L3 switching entity of aswitching module decides to multicast a packet, hashes the packet todetermine the packet direction on one of the two interfaces 602 andsends the packet as a multicast packet. Each switching module receivingthe packet copies the packet to its L2/L3 switching entity for furtherswitching to their customer port(s) and source customer and performsMAC/VLAN learning with the originating switching module, withoutstripping the packet off the ring. Thereafter, the final receivingswitching module or the originating switching module strips the packetfrom the ring. If during transmission of the packet, one of thereceiving switching modules malfunctions, the sending switching modulere-steers th packet through its MAC interfaces 602.

In this embodiment, to ensure the fairness principle, for rateprovisioned packet flows, local traffic uploading should be guaranteedin the presence of pass-through traffic. A congestion status is detectedand advertised to all upstream switching modules when a switchingmodule, in the ring topology, is unable to upload local traffic for aconsistent period due to too much pass-through traffic. Once notifiedabout a congestion station, each upstream switching module reduces itsupload shaping rate accordingly so that the congested switching modulehas a chance to upload its traffic. As an optimization point, traffic tothe switching modules prior to the congested switching module is notaffected unless a prior congestion point is detected.

Every switching fabric on the ring is assigned a unique stationidentifier. One embodiment of the invention allows up to 256 switchingfabrics on the ring. Ethernet packet encapsulation is enhanced withexplicit tag information in place of preamble fields. Specifically, thering header structure is designed to include a start of logical linkdelimiter, a type field for packet/message type differentiation, amulticast indication, a next hop count for ring transmission scopelimiting, a destination switching fabric identifier for packet/messagetarget(s) identification, a source switching fabric identifier forpacket/message originator identification and an error correction field.Multiple virtual MAC service interfaces are presented to the MAC clientlayer. In an embodiment, up to 256 virtual unicast MACs and onemulticast MAC is present at each MAC instance. Each virtual unicast MACpresents a dedicated flow control interface to the MAC client layerthrough the corresponding MAC control sub-layer. Traffic flows on thering are divided into rate provisioned and non-rate provisioned. Forrate provisioned traffic flows the rate is reserved over every linkalong the path from a source switching fabric to a destination switchingfabric. For example, control messages are considered rate provisioned.For non-rate provisioned traffic flows the rate is not reserved acrossthe ring path. The rate traffic is regulated automatically through flowcontrol mechanisms designed for fair access of the ring bandwidth leftover by the rate provisioned traffic.

With respect to the present invention, network devices may be any devicethat utilizes network data, and can include switches, routers, bridges,gateways or servers. In addition, while the above discussionspecifically mentions the handling of packets, packets, in the contextof the instant application, can include any sort of datagrams, datapackets and cells, or any type of data exchanged between networkdevices.

The foregoing description has been directed to specific embodiments ofthis invention. It will be apparent, however, that other variations andmodifications may be made to the described embodiments, with theattainment of some or all of their advantages. Therefore, it is theobject of the appended claims to cover all such variations andmodifications as come within the true spirit and scope of the invention.

What is claimed:
 1. A network device for implementing a high speedtransmission protocol, the network device comprising: a plurality ofhigh speed modules which are connected by a plurality of high speedlinks, each of the plurality of high speed modules implementing the highspeed transmission protocol and each of the plurality of high speedmodules being implemented in an Ethernet ring topology, wherein a mediumaccess control layer in each of the plurality of high speed modules ismodified such that the medium access control layer is configured toseparately queue (i) packets communicated between the high speed modulesof the Ethernet ring, (ii) packets entering the Ethernet ring and (iii)packets leaving the Ethernet ring, wherein the modification istransparent to a switching entity in each of the plurality of high speedmodules and the medium access control layer is configured to communicatepackets from at least two of the separate queues over a same high speedlink, and the high speed transmission protocol comprises a plurality ofaspects including an in-band messaging mechanism for efficient andresponsive traffic management and network operation.
 2. The networkdevice according to claim 1, wherein each of the plurality of high speedmodules is configured to include an access component comprising: a dualmedium access control interface configured as a single trunk interfaceto a network media; a copying and striping control component configuredto filter received packets for downloading and transition forwarding; aplurality of download queues configured to queue the packets leaving theEthernet ring that are to be processed by the switching entity; aplurality of transition queues configured to queue the packetscommunicated between the high speed modules of the Ethernet ring; acongestion and topology management entity for handling protocols on ringcongestion and flow control and ring topology configuration and statuschange notification; a plurality of upload queues configured to queuethe packets entering the Ethernet ring that are received from theswitching entity; and a fair access transmission scheduler for handlingarbitration between uploading and transitional packets and for steeringpackets between the dual medium access control interface.
 3. The networkdevice according to claim 2, wherein the dual medium access controlinterface in each of the plurality of high speed modules is configuredto handle encapsulation and error control for packet transmission. 4.The network device according to claim 2, wherein each of the pluralityof high speed modules is configured to randomly select a direction onthe dual medium access control interface on which to transmit a packet.5. The network device according to claim 1, wherein the network deviceis configured for unicast forwarding such that there is a full-duplexlogical link between every pair of ring switching peers.
 6. The networkdevice according to claim 1, wherein the network device is configuredfor multicast forwarding such that there is a multi-drop link for a ringswitching module to all of associated ring switching module peers. 7.The network device according to claim 1, wherein the network device isconfigured for rate-provisioned packet flows, such that local trafficuploading is guaranteed in the presence of pass-through traffic.
 8. Thenetwork device according to claim 1, wherein the network device isconfigured such that congestion status is detected and advertised to allupstream high speed modules when one of the plurality of high speedmodules is unable to upload local traffic for a consistent period due topass-through traffic.
 9. The network device according to claim 1,wherein the network device is configured to assign a unique stationidentifier to each of the plurality of high speed modules.
 10. Thenetwork device according to claim 1, wherein the network device isconfigured such that Ethernet packet encapsulation is enhanced withexplicit tag information in place of preamble fields.
 11. A high speedmodule implementing a high speed transmission protocol in a networkdevice, the module comprising: means for connecting to a plurality ofhigh speed modules by a plurality of high speed links, each of the highspeed modules being implemented in an Ethernet ring topology; and meansfor modifying a medium access control layer such that the medium accesscontrol layer is configured to separately queue (i) packets communicatedbetween the high speed modules of the Ethernet ring, (ii) packetsentering the Ethernet ring and (iii) packets leaving the Ethernet ring,wherein the modification is transparent to a switching entity in each ofthe plurality of high speed modules and the medium access control layeris configured to communicate packets from at least two of the separatequeues over a same high speed link, wherein the high speed transmissionprotocol comprises a plurality of aspects including an in-band messagingmechanism for efficient and responsive traffic management and networkoperation.
 12. The module according to claim 11, wherein each of thehigh speed modules is configured to include an access componentcomprising: a dual medium access control interface configured as asingle trunk interface to a network media; a copying and stripingcontrol component configured to filter received packets for downloadingand transition forwarding; a plurality of download queues configured toqueue the packets leaving the Ethernet ring that are to be processed bythe switching entity; a plurality of transition queues configured toqueue the packets communicated between the high speed modules of theEthernet ring; a congestion and topology management entity configured tohandle protocols on ring congestion and flow control and ring topologyconfiguration and status change notification; a plurality of uploadqueues configured to queue the packets entering the Ethernet ring thatare received from the switching entity; and a fair access transmissionscheduler configured to handle arbitration between uploading andtransitional packets and for steering packets between the dual mediumaccess control interface.
 13. The module according to claim 12, whereinthe dual medium access control interface in each of the high speedmodules is configured to handle encapsulation and error control forpacket transmission.
 14. The module according to claim 12, wherein eachof the high speed modules is configured to randomly select a directionon the dual medium access control interface on which to transmit apacket.
 15. The module according to claim 11, wherein the module isconfigured for rate-provisioned packet flows, such that local trafficuploading is guaranteed in the presence of pass-through traffic.
 16. Themodule according to claim 11, wherein the module is configured such thatcongestion status is detected and advertised to all upstream high speedmodules when one of the plurality of high speed modules is unable toupload local traffic for a consistent period due to pass-throughtraffic.
 17. The module according to claim 11, wherein the module isconfigured to assign a unique station identifier to each of the highspeed modules.
 18. The module according to claim 11, wherein the moduleis configured such that Ethernet packet encapsulation is enhanced withexplicit tag information in place of preamble fields.
 19. A method forimplementing a high speed transmission protocol in a network device, themethod comprising: connecting to a plurality of high speed modules by aplurality of high speed links, each of the high speed modules beingimplemented in an Ethernet ring topology; and modifying a medium accesscontrol layer of each of the high speed modules such that the mediumaccess control layer is configured to separately queue (i) packetscommunicated between the high speed modules of the Ethernet ring, (ii)packets entering the Ethernet ring and (iii) packets leaving theEthernet ring, wherein the modification is transparent to a switchingentity in each of the plurality of high speed modules and the mediumaccess control layer is configured to communicate packets from at leasttwo of the separate queues over a same high speed link, wherein the highspeed transmission protocol comprises a plurality of aspects includingan in-band messaging mechanism for efficient and responsive trafficmanagement and network operation.